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Executive Summary 



In September 2005, the New York City Department of Youth and Community 
Development (DYCD) launched the Out-of-School Time Programs for Youth (OST) initiative to 
provide young people throughout New York City with access to high-quality programming after 
school, on holidays, and during the summer at no cost to their families. Working closely with 
the nonprofit community and with ten City agencies, including the Department of Education 
(DOE), DYCD created a comprehensive public system of out-of-school time programming. 

Each OST program is operated by a nonprofit organization and is located in a school, community 
center, settlement house, religious center, cultural organization, library, or a public housing or 
Parks facility. 

DYCD contracted with Policy Studies Associates (PSA) to conduct a comprehensive 
evaluation of the OST initiative. PSA designed the evaluation to describe the characteristics of 
OST programs and participants and to assess the initiative’s contribution to the growth and 
development of New York City youth. In addition to providing descriptive information on the 
initiative and its participants, the third-year report explores the associations among program 
quality, patterns of youth participation, and youth outcomes, using known features of high- 
quality programs as identified through previous research. 



Scope of Services 

Since its inception in 2005, the OST system has provided services to a total of 181,001 
individual youth throughout New York City. During the 2007-08 school year, more than 81,000 
youth participated in one of 622 OST programs citywide, all of which were available at no cost 
to families. Elementary-grades programs generally provided services from 3 to 6 pm Monday 
through Friday. Middle-grades programs offered at least eight hours of programming per week, 
and high school programs offered a minimum of 108 hours of programming on a flexible 
schedule throughout the school year. In addition, in order to meet the needs of working families, 
programs serving elementary- and middle-grades youth were open on school holidays. Many 
programs also offered programming for 50 hours per week for eight weeks during the summer. 

The OST initiative grew over its first three years to serve increasingly large numbers of 
youth each year. In the first year of the initiative (2005-06), 50,618 youth enrolled in OST 
programs throughout the city, including 40,584 participants in the Option I programs that are at 
the core of this report. In the second year of the initiative (2006-07), this number had increased 
to 68,449 participants overall and 56,742 in Option I, and by the third year the initiative had 
grown to serve 81,213 youth, including 67,524 Option I participants. The initiative also 
expanded in terms of the number of programs offering OST services, with the addition of 1 1 1 
new elementary-grades programs in the third year of the initiative. 
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Program Quality 



Varied program activities. Aligned with core program goals, OST programs generally 
offered varied activities, including academic skills enhancement, cultural exposure and 
enrichment, sports, recreation, community service, and leadership development. Nearly all 
elementary- and middle-grades programs offered academic enhancement, arts and culture, and 
recreational activities. About two-thirds of these programs offered life-skills activities, and 
about half offered community-building activities. Consistent with the age ranges they serve, 
relatively few elementary- and middle-grades programs offered any career or work activities in 
the third year of the OST initiative (15 percent and 24 percent, respectively). In contrast, high 
school OST programs often focused on a narrow set of activities or topic areas. Reflecting that 
focus, the activity patterns in high school programs differed from those of elementary- and 
middle-grades programs: while more than two-thirds of high school programs offered academic 
enhancement, arts and culture, and recreational activities, only about half of high school 
programs offered life skills, community building, and career or work activities. 

Exposure to new experiences. Overall, participants reported a high degree of satisfaction 
with the extent to which their OST program exposed them to new experiences (average score of 
3.20 out of 4 on the youth-survey scale). Elementary-grades youth were somewhat more likely 
to report a high level of participation in new experiences (3.34 out of 4) than were middle-grades 
(3.22) and high school youth (3.27). Based on the evaluation’s statistical measure of the 
difference in responses, high school youth responded notably more positively to this scale than 
did middle-grades youth, suggesting that in general there is most room for improvement in 
creating engaging, innovative activities at the middle-school level. 

Observed program quality. In structured observations of program activities in a 
stratified, random sample of 15 programs, evaluators found that OST staff generally created 
warm, welcoming, yet structured environments. However, programs struggled in their efforts to 
implement active learning opportunities through activities that built on each other in a sequenced 
manner. On average, observed activities demonstrated clear activity goals (5.07, out of 7 on the 
evaluation’s task-oriented scale). Activities also rated high on the relationship-focused scale, 
which measures the extent to which activities developed personal and social skills (4.21, out of 
7). In contrast, evaluators observed less evidence of activity quality as defined by the extent to 
which the activity built on skills and content previously learned (3.60, out of 7, on the skill- 
building scale) or engaged youth in active, hands-on learning experiences (1.75, out of 7, on the 
active-learning scale). 



Positive Relationships Involving Youth 

Youth interactions with peers. Y outh were overwhelmingly positive in their assessment 
of their social interactions within the program (average scale score of 3.32, out of 4), although 
both middle-grades and high school youth responded somewhat more positively than did 
elementary-grades participants (3.35 and 3.37, compared to 3.28). These high scores are an 
indication that OST programs were successful in creating welcoming social environments for 
youth to develop friendships. 
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Youth interactions with staff. OST participants also responded very positively to a scale 
measuring the quality of their interactions with program staff (average scale sore of 3.35, out of 
4), with high school youth responding somewhat more positively than either elementary-grades 
or middle-grades youth (3.46, compared to 3.31 for both elementary- and middle-grades youth). 



Effective Program Partnerships and Supports 

Staffing patterns. The majority of programs (82 percent) hired college students, and 69 
percent of programs hired at least one professional specialist (e.g., a professional artist, coach, 
dancer). In addition, 63 percent of programs had at least one certified teacher on staff, and 61 
percent hired teen staff members. Fifty percent of OST program directors reported that a staff 
member was assigned to be a master teacher or educational coordinator within the program. 

OST programs were strategic in the roles they assigned to certified teachers and 
specialists within the programs. Certified teachers were employed mainly to provide academic 
support to programs by leading academic activities (72 percent of programs) and tutoring (7 1 
percent). Specialists were hired primarily for non-academic enrichment activities such as arts 
and sports (88 percent). College students, in contrast, played roles across program activities, 
including tutoring youth (88 percent) and assisting with enrichment and academic activities (82 
percent and 78 percent, respectively). More than two-thirds of programs also employed college 
students to lead enrichment and academic activities (73 percent and 70 percent, respectively). 
Teen staff were mostly hired as activity assistants (82 percent) or as tutors (80 percent). Through 
these mixed staffing patterns, programs aimed to involve young staff members in connecting 
with younger participants; teen staff were supported by experienced staff who could provide 
ongoing mentoring and supervision. Thanks to these varied staffing patterns, including the use 
of both paid and volunteer staff, OST programs maintained an overall low ratio of youth-to-staff, 
with a median ratio of 8: 1 . 

Building staff capacity. With the launch of the OST initiative in 2005, DYCD contracted 
with the Partnership for After-School Education (PASE) to provide technical assistance and 
professional development workshops for OST program staff. In survey responses, 88 percent of 
program directors reported that PASE was a primary source of technical assistance and training 
for themselves and their staff members. Other primary sources of technical assistance included 
the provider organizations (67 percent) and the DYCD program manager (40 percent). Program 
directors reported that their staff received professional development through staff meetings at the 
program (86 percent), internal staff orientations (66 percent), and off-site workshops (62 
percent). 

Partnerships with schools. In the third year of the OST initiative, program directors 
reported regular communication with school staff in several areas. Program directors reported 
communicating with school administrators or staff at least monthly about: the needs or progress 
of individual students (61 percent), issues related to sharing classroom space (56 percent), 
homework assignments (56 percent), and student discipline policies (50 percent). In surveys, 
directors reported high levels of satisfaction with their communication with school staff. Sixty- 
one percent of program directors said that receiving responses to requests to coordinate services 
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or resources with school staff was not a challenge; only 6 percent reported that this was a major 
challenge. 



Partnerships with parents. Programs reached out to the families of participants to 
engage them and to meet youth needs more effectively. Nearly all program directors (91 
percent) had conversations with parents over the phone at least a few times a month; 83 percent 
met in person with parents that frequently. Programs also relied on family or parent liaisons to 
engage families and encourage high rates of participation: 45 percent of programs employed 
someone for this role. 



Evidence of Youth Outcomes 

Program engagement. In the third year of the OST initiative, programs on average 
exceeded their targeted enrollment levels. Option I programs had a target enrollment overall of 
approximately 63,000 youth, based on the contracts awarded by DYCD; programs actually 
served a total of about 64,500 students from September 2007 through June 2008. Even so, some 
individual programs could not meet their targeted enrollment, measured by the number of slots 
available for participants as established in the program’s contract with DYCD. Sixty-nine 
percent of elementary programs met or exceeded their enrollment targets, as did 70 percent of 
middle-grades programs and 59 percent of high school programs. 

As in Years 1 and 2, programs reached a high standard of participation. On average, 
elementary-grades participants attended 377 hours during the year, compared to the 432 hours 
they were expected to attend. This represents an average of 87 percent of targeted hours, 
exceeding average participation rates of 72 percent and 83 percent attained in Year 1 and 2, 
respectively. Older youth also surpassed their targeted number of OST participation hours, on 
average. Middle-grades participants as a group achieved their targeted number of hours of 
participation: on average, middle-grades participants attended 218 hours of the 216 hours 
expected at the middle-grades level (101 percent of the targeted hours). High school participants 
exceeded their targeted number of hours of participation, attending on average 92 hours in the 
third year of the initiative, 16 hours above their target of 76 hours. 

The evaluation also measured the duration of participation across years: 6,371 youth 
participated in all three years of OST school-year programming. Almost 22,000 youth 
participated in two years of OST programming, either in Years 1 and 2 or in Years 2 and 3, while 
102,837 participated in a single year of programming (Year 1, Year 2 or Year 3). Approximately 
22,000 youth only participated in summer programming. 

Social development. Overall, participants reported a strong sense of belonging in the 
third year of the initiative, with an average youth-survey scale score of 3.38, out of 4. In 
particular, more than two-thirds of participants “agreed a lot” that they felt safe in the program 
(68 percent) and 58 percent said that the program was a “good place to hang out.” High school 
students were especially likely to report a strong sense of program connection (scale score of 
3.48, compared to 3.32 for elementary-grades participants and 3.38 for middle-grades 
participants). 
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Educational development. Analyses examined the relationship between OST 
participation and the following measures of educational development: academic motivation, 
school attendance rate, academic benefits reported by youth, performance on the state English 
language arts (ELA) and mathematics tests (for grades 3-8), and credits accrued and Regent 
exams passed for high school students. 

As in prior years, participants reported an overall strong level of academic motivation in 
Year 3, with an average scale score of 3.34 out of 4. Elementary-grades participants reported 
notably higher levels of academic motivation than did middle-grades or high school participants 
(scale score of 3.34, compared to 3.25 for both groups of older youth). 

Participants reported moderate levels of academic benefits on a survey scale (average 
scale score of 3.06, out of 4). The most common academic benefit reported by participants was 
that the program helped them to finish their homework more often (54 percent agreed a lot). 
However, no significant differences were found between participants and matched 
nonparticipants on the measures of educational perfonnance that are maintained by the 
Department of Education. OST participants and nonparticipants in the 15 sampled sites were 
closely matched on their baseline academic perfonnance in both the state ELA and math tests. 
Consistent with citywide trends, both groups showed small improvements in performance over 
the course of the OST initiative, with no significant differences in the size of the gains posted by 
the two groups. For high school youth in the sampled sites, evaluators examined the cumulative 
number of course credits accrued after each year of OST participation. Analyses found no 
significant differences between the groups on this measure. Analyses also examined the number 
of New York State Regents exams that participants and matched nonparticipants had taken and 
passed to assess progress towards graduation, and found no significant differences. Analyses of 
school-attendance patterns of participants and matched nonparticipants found no notable 
differences in attendance changes over time. 



Relationships Among Participation, Program Quality, and Youth Outcomes 

Relationship between program quality and youth outcomes. Overall, correlation 
analyses revealed positive relationships between youth reports of the extent to which a program 
exposed youth to new experiences and youth reports of their sense of belonging in the OST 
program, their academic benefits, and their rates of school attendance. However, a program’s 
mean rating of exposing youth to new experiences was negatively correlated with individual 
youths’ mean hours of participation in OST. A possible explanation for this pattern is that youth 
who attend OST more become accustomed to the program offerings over time and therefore 
lower their perceptions of novelty within the program. The breadth of OST program content in 
OST programs, measured by the number of different activity types offered, was positively 
associated with the total number of hours of youth participation and with the number of credits 
earned by high school participants. 

Relationships involving social interactions. In general, measures of a supportive OST 
environment, including average reports of youth interactions with their peers and with staff 



5 




members in the program, were positively correlated with youth reports of their sense of 
belonging and of the program’s academic benefits. 

Relationships involving effective partnerships and supports. Several measures of 
effective partnerships and supports were negatively associated with youth outcomes. For 
example, the number of professional development opportunities in which OST staff members 
participated was negatively correlated with youth program participation, youth reports of sense 
of belonging, and school attendance. Rather than being an indication that staff professional 
development is ineffective, more likely this finding is an indication that programs that are 
struggling to implement a high-quality program and contribute to positive youth outcomes are, in 
fact, taking greatest advantage of the professional development opportunities available through 
DYCD, PASE, and other resources, either as a result of a referral to technical assistance by their 
DYCD program manager or by their own choice. Measures of communication with schools and 
with parents were positively associated with the number of hours of youth participation in OST 
programming. 

Program quality index. Based on all of the findings from the correlation analyses, 
evaluators created a program quality index as a tool for assessing the overall quality of an OST 
program. Final components of the program quality index include exposure to new experiences, 
youth interactions with peers, youth interactions with staff, wide mix of staff, presence of a 
master teacher, and presence of a parent liaison. Analyses found a positive relationship between 
the program quality index and whether the program succeeded in meeting its targeted enrollment 
level. Analyses also found positive correlations between overall program quality and aggregate 
youth reports of their sense of belonging in the program, their engagement in pro-social 
behaviors, their academic motivation, and their academic benefits. Multi-level analyses revealed 
some significant relationships between OST program quality and youth outcomes, even though 
program-level variables accounted for only a small amount of variance in youth outcomes. 



Creation of a City-wide OST System 

Capacity of provider organizations. Executive directors of provider organizations 
reported that the OST initiative increased their organizational capacity in several ways. More 
than half of executive directors reported that the initiative increased the organization’s capacity 
to reach out to serve more youth and families to a great extent or somewhat (83 percent), provide 
staff training and technical assistance (73 percent), partner with a public school (71 percent), 
partner with cultural organizations (65 percent), partner with city agencies (63 percent), offer 
programming on weekends and holidays (59 percent), and provide a career ladder for OST staff 
(57 percent). 

Executive directors also reported increased capacity to implement certain practices in the 
third year of the OST initiative, compared to previous years. Fifty-nine percent of executive 
directors reported that their programs provided much more or somewhat more training and 
technical assistance for staff in the third year of the initiative than in the first or second year. 
Overall, executive directors reported high levels of satisfaction with DYCD’s management of the 
OST initiative and support of OST. More than half of executive directors reported that they were 
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very satisfied with the support provided by the DY CD program manager (67 percent) and the 
opportunities for staff professional development (55 percent). Executive directors were least 
satisfied with DYCD’s assistance in negotiating partnerships with schools (29 percent reported 
that they were very satisfied). 

Meeting the needs of working families. A goal of the OST initiative was to provide 
support to working families in New York City, particularly in the target zip codes identified as 
priorities for out-of-school time services. Survey responses from parents of OST participants in 
the elementary- and middle-grades indicated that overall the initiative succeeded in reaching this 
goal and meeting the needs of families: about three-quarters of parents rated the OST program 
that their child attended as either excellent (43 percent) or very good (33 percent). 

Parents especially valued the academic-support features of OST programs. Forty-seven 
percent of parents cited homework help as the most important activity in the after-school 
program, and an additional 26 percent cited academic enrichment as the most important activity. 
In addition, parents’ reports of their reasons for enrolling their child in the OST program 
reflected an emphasis on seeking academic support: 76 percent believed the program would help 
their child do better in school, and 72 percent wanted their child to get help with homework. 
Seventy-five percent of parents also said that they enrolled their child in an OST program to 
provide them with the opportunity to participate in new activities. 

Parents also responded positively to questions about the ways in which the OST initiative 
had enabled them to work more or pursue more education. Across all responding parents, 74 
percent agreed that the program made it easier for them to keep their job, and 73 percent agreed 
that they miss less work than they had previously because their children attended the OST 
program. In addition, 7 1 percent of parents reported that they were able to work more hours 
because their children were in the program. 



Looking Ahead to Long-Term Sustainability 

Through the first three years of the OST initiative, DYCD has established foundations in 
policy and practice for the long-term sustainability of high-quality, publicly funded, out-of-school 
time programming for the youth of New York City. With the launch of the OST initiative in 2005, 
the collaborative relationship between DYCD and the New York City Department of Education 
has grown, particularly through adoption of a Memorandum of Understanding committing 
hundreds of public schools as sites for OST programs and the provision of in-kind support by the 
DOE for OST programs through facilities, security, and snacks and meals for OST participants. At 
the same time, DYCD strengthened its network of community -based organizations, foundations, 
and providers of technical assistance to support the initiative through partnerships with the Wallace 
Foundation and the Partnership for After-School Education, among others. Importantly, the City 
has included funding for OST programming in its four-year financial plan and the budget for OST 
programming has steadily increased from $46.4 million in FY 2006 to $1 17.1 million in FY 2009. 

As the OST initiative enters its next phase, evaluation findings from the first three years 
point to elements of program quality to maintain as well as possible areas in which focused 
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resources and technical assistance can improve quality and youth outcomes. DYCD has already 
committed resources to improving the quality of its monitoring and support of OST programs 
through ongoing technical assistance opportunities, particularly those focused on data 
management, behavior management techniques, and program content development. The DY CD 
Online data management system offers opportunities to continue to track patterns of program and 
activity engagement across the initiative and within specific programs. The development of new 
program management tools, including those that track the implementation of features in the 
evaluation’s program quality index, will provide opportunities to continue to strengthen the 
capacity of OST programs to provide high-quality services to youth. 

Additional research-based steps will support the city’s OST providers as they shape the 
city’s OST system of the future. These steps include the following: 

■ Assist programs in identifying resources — or in learning to better plan and budget 
existing resources — directed specifically to hiring specialized staff members to 
maximize youth recruitment and engagement (e.g., parent liaisons) and to help 
plan and oversee high-quality, structured program content (e.g., certified teachers 
or professional specialists). 

■ Focus technical assistance related to activity planning on teaching staff strategies 
to engage youth in dynamic, active learning opportunities in which they discuss, 
collaborate, plan, and take on leadership roles, regardless of the content area. 

■ Through technical assistance, encourage OST programs and provider 
organizations to utilize the capacity of the DYCD Online system to generate data 
that can support program management and improvement efforts, including, for 
example, the monitoring of program participation patterns to determine whether 
certain types of activities appeal more or less to particular groups of students. 
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